Video Configuration

The easiest way to configure the webrtc-capabilities is to use the UI in the Transitive Portal. It allows you to choose how many streams you want, and for each stream allows you to pick the video source, plus any additional source-specific parameters, e.g., resolution and frame rate on v4l devices (i.e., USB camera), or the topic for ROS sources. Separately you can choose to enable audio, configure the bitrate, and enable debug options if necessary.

Once configured, the component will establish a peer-to-peer WebRTC connection with your device and start showing the video. At this point you can click the "Embed" link to retrieve a React or HTML snippet for embedding the configured video stream(s) in your own web pages.

Dynamic Bitrate

In the embedding code you can set a bitrate for the streaming session. Changing this value dynamically in React will end the current session and start a new one with the new bitrate.

If you want to dynamically change the bitrate without restarting the session instead, you can set the dynamic_bitrate property to a number indicating KB/s. This property, unline bitrate, will maintain the ongoing session and just update the maximum bitrate dynamically instead. This allows you to implement features such as dynamic switching between "SD" and "HD" mode, or similar.

Note that whatever bitrate you set, statically or dynamically, will only be the maximum bitrate used. Congestion control may still reduce the actual streaming bitrate to account for network issues, unless you explicitly disable it or use pre-encoded video sources, which don't support congestion control.

Hardware Accelerated h264 Encoding

The capability will automatically use any available and supported hardware encoders on your device. Currently supported are NVIDIA boards like Jetsons and Orins, NVIDIA Deepstream 6.3+, Rockchip based platforms like the Orange Pi and Firefly, and Intel graphics chipsets that support VA-API.

Note: Bandwidth optimization is not yet implemented for Rockchip hardware encoders. Please let us know if you are using a Rockchip based platform. This will help us prioritize adding support for this if you need it.

NVIDIA

On Jetson platforms there is nothing you need to do to enable hardware acceleration. These platforms come with the necessary drivers and libraries pre-installed.

Intel VA-API

If your compute platform supports hardware encoding via the VA-API, WebRTC Video will automatically utilize it if the following requirements are met:

• The apt packages gstreamer1.0-vaapi, libdrm-dev, libva-dev, and i965-va-driver (or equivalent driver for your hardware) are installed.
• These environment variables are set:

  GST_VAAPI_ALL_DRIVERS=1
  LIBVA_DRIVER_NAME=iHD # or replace with the name of your driver, e.g., i965
  LIBVA_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri
  

  One way to set these environment variables is by adding them to ~/.transitive/.env_user (create the file if it doesn't exist).

Using VA-API in Docker

• Install gstreamer1.0-vaapi inside the container.
• Install libdrm-dev, libva-dev, and i965-va-driver (or equivalent driver for your hardware) both inside and outside the container
• Add these arguments to your docker run command (or corresponding fields in docker-compose.yaml):

  --device=/dev/dri/
  -e GST_VAAPI_ALL_DRIVERS=1
  -e LIBVA_DRIVER_NAME=i965 # or replace with the name of your driver, e.g., iHD
  -e LIBVA_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri
  

Deciding on a bitrate

In order to find an appropriate bitrate for your streams, we suggest 0.02 bytes per pixel as a rule of thumb. For example, one stream of 640x480 at 15 fps results in 4,608,000 pixels per second, multiplied by 0.02 bytes/pixel gives 92 KB/s as a suggested bitrate. You can set it lower if necessary, but the quality will really suffer. Of course, if your connection allows it, you can always set it higher to increase image quality.

As of v0.22.2 the capability features an auto-bitrate mode which applies the above rule of thumb automatically when no bitrate is specified explicitly.

Framerate

When using a v4l source, i.e., a USB camera, you can choose any of the framerates provided by the hardware. When using ROS topics, the default framerate is 15/1, i.e., 15 frames per second. This can be changed by providing the framerate option in the embedding code, e.g., framerate="10/1".

RTSP streams (IP cameras)

To use an RTSP stream as video source, such as those provided by many IP cameras, set the type in your embedding HTML to rstp, and set the source to the RTSP URL provided by your camera. For instance:

  <webrtc-video-device id="superbot" host="transitiverobotics.com" ssl="true"
    jwt={jwt}
    count="1"
    timeout="1800"
    type="rtsp"
    source="rtsp://my-ip-camera:8554/cam"
    />

This assumes that your stream is already h264 encoded, as is common with IP cameras, and that you do not wish to transcode the stream. This is by far the most CPU efficient option but also implies that the webrtc-video capability will not be able to provide congestion control or set the bitrate. If you do want to decode the stream and re-encode it in order to get congestion control and bitrate control back, you can use type="rtsp-transcode" instead.

ROS topics carrying pre-encoded h264 streams

While we do not recommend this, some users may choose to multiplex their h264 video streams via ROS topics. ROS does not have a standard message format for video streams, but it is possible to "abuse" any ROS message type with a binary array data type, e.g., sensor_msgs/CompressedImage.

Assuming you have such a stream in a topic called /camera/h264, you can use that as your video input source in embedded code like this:

<webrtc-video-device id="superbot" host="transitiverobotics.com" ssl="true"
  jwt={jwt}
  timeout="1800"
  count="1"
  rosversion="1"
  type="rostopic-h264"
  source="/camera/h264"
/>

or in React:

<TransitiveCapability jwt={jwt} timeout="1800" count="1"
  rosversion="1" type="rostopic-h264" source="/camera/h264" />

where jwt, as always, is a JSON Web Token carrying the payload described in the Embed instructions of this capability on the portal. For ROS 2 just change the rosversion.

Note that whenever you use a pre-encoded h264 streams as input, you lose a very significant and important feature: congestion control. Hence, if your input stream uses a higher bitrate than your network can support, there will be no way for webrtc-video to adjust the bitrate to ensure low-latency. We therefor recommend against using pre-encoded h264 streams. When you do use them, please make sure they use a bitrate that is appropriate for your Internet connection.

Custom Video Source Pipelines

In addition to various video sources, the capability supports the specification of custom GStreamer source pipelines. In your embedding HTML you can specify type custom and set as source your custom pipeline. For example:

  <webrtc-video-device id="superbot" host="transitiverobotics.com" ssl="true"
    jwt={jwt}
    count="1"
    timeout="1800"
    type="custom"
    source="videotestsrc is-live=true ! video/x-raw,framerate=(fraction)15/1,width=640,height=480"
    />

This assumes that the sink of your pipeline can be fed into a videoconvert element for conversion to video/x-raw. If your pipeline produces a stream that is already h264 encoded and you don't want to decode and re-encode this stream, then you can set the type to custom-h264. Note that in that case the capability will not implement any congestion control for you.

This is an advanced feature and only meant for users who are familiar with GStreamer pipelines. It is also more difficult to debug. We recommend to test your custom source pipeline first using gst-launch-1.0 and autovideosink or fakesink.

Custom Video Sink Pipelines

In many cases, streaming video is only one of several usages of a robot's video sources. When this is the case it is useful to be able to "tee" the stream at several locations in the pipeline to feed it into auxiliary processing pipelines. Local recording is an example of this, but a very specific one. To support any other such applications, webrtc-video gives the ability to specify additional sink pipelines that can be injected after the encoding step of the pipeline. This is done per-stream.

Example

  <webrtc-video-device id="superbot" host="transitiverobotics.com" ssl="true"
    jwt={jwt}
    bitrate=50
    count="2"
    timeout=1800
    type="v4l2src"
    source="/dev/video0"
    streamtype="image/jpeg"
    framerate="15/1"
    width="640"
    height="480"
    encodedpipe="splitmuxsink location=/tmp/video0.mp4 max-size-time=60000000000 max-files=10"
    type_1="v4l2src"
    source_1="/dev/video2"
    streamtype_1="image/jpeg"
    framerate_1="15/1"
    width_1="640"
    height_1="480"
    encodedpipe_1="splitmuxsink location=/tmp/video1.mp4 max-size-time=60000000000 max-files=10"
    />

Local Recording

The capability supports recording all outgoing video on disk in a rolling buffer. This feature, similar to black-box recorders on airplanes, can be useful when investigating recent incidences after the fact. The buffer restarts each time a new session is started and currently records a maximum of ten minutes or 1 GB, whichever comes first. Add record="true" to your HTML embedding code to enable this. The recordings will be in /tmp/stream_*.mp4.

Device Sharing

The camera streams on robots are often used for multiple purposes, remote streaming only being one of them. When this is the case, the stream needs to be shared. This is trivial for ROS topics and RTSP streams, but not for USB cameras (video4linux2 devices). This is because under Linux only one process can open such a device at a time. Fortunately there are a few options for sharing the streams from these devices.

RTP over UDP

In this approach we feed the stream coming from the camera to multiple UDP sockets using the RTP protocol. These UDP sockets can then be accessed using gstreamer's udpsrc element.

# producer
gst-launch-1.0 -e v4l2src device=/dev/video2 ! video/x-h264,width=640,height=480,framerate=30/1 ! h264parse config-interval=-1 ! rtph264pay ! multiudpsink clients=127.0.0.1:9001,127.0.0.1:9002
# Client 1
gst-launch-1.0 udpsrc port=9001 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! autovideosink
# Client 2
gst-launch-1.0 udpsrc port=9002 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! autovideosink

Shared Memory (recommended)

Similar to the UDP approach, we can use shared memory to share the stream:

# Producer
gst-launch-1.0 -e v4l2src device=/dev/video4 ! video/x-raw,width=640,height=480,framerate=30/1 ! shmsink socket-path=/tmp/foo2 shm-size=2000000 wait-for-connection=false sync=true
# Clients
gst-launch-1.0 shmsrc socket-path=/tmp/foo2 do-timestamp=true is-live=true ! video/x-raw,width=640,height=480,framerate=30/1,format=YUY2 ! autovideosink

Or, for a h264 camera stream:

# producer
gst-launch-1.0 -e v4l2src device=/dev/video2 ! h264parse config-interval=-1 ! shmsink socket-path=/tmp/foo shm-size=2000000 wait-for-connection=false sync=true
# consumer
gst-launch-1.0 shmsrc socket-path=/tmp/foo do-timestamp=true is-live=true ! 'video/x-h264,profile=baseline,framerate=30/1' ! h264parse config-interval=-1 ! decodebin ! autovideosink

Virtual devices (v4l2loopback; not recommended)

The v4l2loopback kernel module allows you to create virtual v4l2 devices which can be shared. Using ffmpeg or gstreamer you can then forward a physical device's stream to one or more such virtual devices.

Example:

# create the virtual devices
sudo modprobe v4l2loopback devices=2 video_nr=10,11 exclusive_caps=1,1 card_label="front_camera,back_camera"
# forward a physical device's stream to virtual device
ffmpeg -f v4l2 -input_format h264 -video_size 640x480 -framerate 30 -i /dev/video3 -c:v copy -f v4l2 /dev/video10

In this example we are forwarding a h264 stream provided by the camera (requires ffmpeg v6+), but the same approach also works for raw and jpeg streams.

Audio support

When running in Docker, you may need to add these parameters to your docker run command (or similarly in docker-compose.yaml) in order for audio to work:

-e PULSE_SERVER=unix:/run/user/1000/pulse/native \
-v /run/user/1000/pulse:/run/user/1000/pulse \
-v ~/.config/pulse/cookie:/root/.config/pulse/cookie \
--group-add $(getent group audio | cut -d: -f3)

It also requires gstreamer1.0-pulseaudio to be installed on robot on older versions of Ubuntu.